{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "tf_seving_colab.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "VDsjbVY8m7Ed",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GriiRKkOnC6O",
        "colab_type": "text"
      },
      "source": [
        "# 6-6-1使用tensorflow-serving部署模型\n",
        "\n",
        "TensorFlow训练好的模型以tensorflow原生方式保存成protobuf文件后可以用许多方式部署运行。\n",
        "\n",
        "例如：通过 tensorflow-js 可以用javascrip脚本加载模型并在浏览器中运行模型。\n",
        "\n",
        "通过 tensorflow-lite 可以在移动和嵌入式设备上加载并运行TensorFlow模型。\n",
        "\n",
        "通过 tensorflow-serving 可以加载模型后提供网络接口API服务，通过任意编程语言发送网络请求都可以获取模型预测结果。\n",
        "\n",
        "通过 tensorFlow for Java接口，可以在Java或者spark(scala)中调用tensorflow模型进行预测。\n",
        "\n",
        "我们主要介绍tensorflow serving部署模型、使用spark(scala)调用tensorflow模型的方法。\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "auj9U6qOnHYu",
        "colab_type": "text"
      },
      "source": [
        "### 〇、tensorflow serving模型部署概述\n",
        "\n",
        "使用 tensorflow serving 部署模型要完成以下步骤。\n",
        "\n",
        "* (1) 准备protobuf模型文件。\n",
        "\n",
        "* (2) 安装tensorflow serving。\n",
        "\n",
        "* (3) 启动tensorflow serving 服务。\n",
        "\n",
        "* (4) 向API服务发送请求，获取预测结果。\n",
        "\n",
        "\n",
        "可通过以下colab链接测试效果《tf_serving》：\n",
        "https://colab.research.google.com/drive/1vS5LAYJTEn-H0GDb1irzIuyRB8E3eWc8\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WLTFOoeRnFHS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "0fe1a46c-e77e-4961-c47e-a1e3f2442b79"
      },
      "source": [
        "%tensorflow_version 2.x\n",
        "import tensorflow as tf\n",
        "print(tf.__version__)\n",
        "from tensorflow.keras import *"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2.2.0-rc2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lsqhU4vtnOws",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6593UJyinRjL",
        "colab_type": "text"
      },
      "source": [
        "### 一，准备protobuf模型文件\n",
        "\n",
        "我们使用tf.keras 训练一个简单的线性回归模型，并保存成protobuf文件。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zPSXDu6_nSa7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "666ac72c-657f-4036-9912-0a2664c3576c"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras import models,layers,optimizers\n",
        "\n",
        "## 样本数量\n",
        "n = 800\n",
        "\n",
        "## 生成测试用数据集\n",
        "X = tf.random.uniform([n,2],minval=-10,maxval=10) \n",
        "w0 = tf.constant([[2.0],[-1.0]])\n",
        "b0 = tf.constant(3.0)\n",
        "\n",
        "Y = X@w0 + b0 + tf.random.normal([n,1],\n",
        "    mean = 0.0,stddev= 2.0) # @表示矩阵乘法,增加正态扰动\n",
        "\n",
        "## 建立模型\n",
        "tf.keras.backend.clear_session()\n",
        "inputs = layers.Input(shape = (2,),name =\"inputs\") #设置输入名字为inputs\n",
        "outputs = layers.Dense(1, name = \"outputs\")(inputs) #设置输出名字为outputs\n",
        "linear = models.Model(inputs = inputs,outputs = outputs)\n",
        "linear.summary()\n",
        "\n",
        "## 使用fit方法进行训练\n",
        "linear.compile(optimizer=\"rmsprop\",loss=\"mse\",metrics=[\"mae\"])\n",
        "linear.fit(X,Y,batch_size = 8,epochs = 100)  \n",
        "\n",
        "tf.print(\"w = \",linear.layers[1].kernel)\n",
        "tf.print(\"b = \",linear.layers[1].bias)\n",
        "\n",
        "## 将模型保存成pb格式文件\n",
        "export_path = \"./data/linear_model/\"\n",
        "version = \"1\"       #后续可以通过版本号进行模型版本迭代与管理\n",
        "linear.save(export_path+version, save_format=\"tf\") "
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "inputs (InputLayer)          [(None, 2)]               0         \n",
            "_________________________________________________________________\n",
            "outputs (Dense)              (None, 1)                 3         \n",
            "=================================================================\n",
            "Total params: 3\n",
            "Trainable params: 3\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/100\n",
            "100/100 [==============================] - 0s 862us/step - loss: 328.7710 - mae: 15.1983\n",
            "Epoch 2/100\n",
            "100/100 [==============================] - 0s 846us/step - loss: 304.3128 - mae: 14.6258\n",
            "Epoch 3/100\n",
            "100/100 [==============================] - 0s 849us/step - loss: 281.1457 - mae: 14.0663\n",
            "Epoch 4/100\n",
            "100/100 [==============================] - 0s 909us/step - loss: 259.4896 - mae: 13.5120\n",
            "Epoch 5/100\n",
            "100/100 [==============================] - 0s 905us/step - loss: 237.5394 - mae: 12.9364\n",
            "Epoch 6/100\n",
            "100/100 [==============================] - 0s 877us/step - loss: 217.2313 - mae: 12.3776\n",
            "Epoch 7/100\n",
            "100/100 [==============================] - 0s 856us/step - loss: 198.5707 - mae: 11.8359\n",
            "Epoch 8/100\n",
            "100/100 [==============================] - 0s 871us/step - loss: 180.6364 - mae: 11.2862\n",
            "Epoch 9/100\n",
            "100/100 [==============================] - 0s 873us/step - loss: 163.5264 - mae: 10.7346\n",
            "Epoch 10/100\n",
            "100/100 [==============================] - 0s 891us/step - loss: 147.0449 - mae: 10.1818\n",
            "Epoch 11/100\n",
            "100/100 [==============================] - 0s 893us/step - loss: 132.0012 - mae: 9.6451\n",
            "Epoch 12/100\n",
            "100/100 [==============================] - 0s 864us/step - loss: 118.1782 - mae: 9.1151\n",
            "Epoch 13/100\n",
            "100/100 [==============================] - 0s 870us/step - loss: 105.0113 - mae: 8.5905\n",
            "Epoch 14/100\n",
            "100/100 [==============================] - 0s 839us/step - loss: 92.7334 - mae: 8.0604\n",
            "Epoch 15/100\n",
            "100/100 [==============================] - 0s 935us/step - loss: 81.5341 - mae: 7.5407\n",
            "Epoch 16/100\n",
            "100/100 [==============================] - 0s 928us/step - loss: 71.1263 - mae: 7.0362\n",
            "Epoch 17/100\n",
            "100/100 [==============================] - 0s 848us/step - loss: 61.6876 - mae: 6.5374\n",
            "Epoch 18/100\n",
            "100/100 [==============================] - 0s 905us/step - loss: 53.1549 - mae: 6.0574\n",
            "Epoch 19/100\n",
            "100/100 [==============================] - 0s 883us/step - loss: 45.6156 - mae: 5.5933\n",
            "Epoch 20/100\n",
            "100/100 [==============================] - 0s 846us/step - loss: 38.4998 - mae: 5.1333\n",
            "Epoch 21/100\n",
            "100/100 [==============================] - 0s 840us/step - loss: 32.4696 - mae: 4.7050\n",
            "Epoch 22/100\n",
            "100/100 [==============================] - 0s 858us/step - loss: 27.0926 - mae: 4.2842\n",
            "Epoch 23/100\n",
            "100/100 [==============================] - 0s 834us/step - loss: 22.3000 - mae: 3.8751\n",
            "Epoch 24/100\n",
            "100/100 [==============================] - 0s 837us/step - loss: 18.2145 - mae: 3.4815\n",
            "Epoch 25/100\n",
            "100/100 [==============================] - 0s 839us/step - loss: 14.5914 - mae: 3.0991\n",
            "Epoch 26/100\n",
            "100/100 [==============================] - 0s 895us/step - loss: 11.7514 - mae: 2.7603\n",
            "Epoch 27/100\n",
            "100/100 [==============================] - 0s 865us/step - loss: 9.3536 - mae: 2.4522\n",
            "Epoch 28/100\n",
            "100/100 [==============================] - 0s 998us/step - loss: 7.6040 - mae: 2.2012\n",
            "Epoch 29/100\n",
            "100/100 [==============================] - 0s 876us/step - loss: 6.3721 - mae: 2.0201\n",
            "Epoch 30/100\n",
            "100/100 [==============================] - 0s 866us/step - loss: 5.6344 - mae: 1.9019\n",
            "Epoch 31/100\n",
            "100/100 [==============================] - 0s 855us/step - loss: 5.1974 - mae: 1.8255\n",
            "Epoch 32/100\n",
            "100/100 [==============================] - 0s 852us/step - loss: 4.9254 - mae: 1.7811\n",
            "Epoch 33/100\n",
            "100/100 [==============================] - 0s 864us/step - loss: 4.7547 - mae: 1.7461\n",
            "Epoch 34/100\n",
            "100/100 [==============================] - 0s 848us/step - loss: 4.5999 - mae: 1.7141\n",
            "Epoch 35/100\n",
            "100/100 [==============================] - 0s 883us/step - loss: 4.4756 - mae: 1.6851\n",
            "Epoch 36/100\n",
            "100/100 [==============================] - 0s 866us/step - loss: 4.3627 - mae: 1.6594\n",
            "Epoch 37/100\n",
            "100/100 [==============================] - 0s 875us/step - loss: 4.2659 - mae: 1.6363\n",
            "Epoch 38/100\n",
            "100/100 [==============================] - 0s 862us/step - loss: 4.1796 - mae: 1.6151\n",
            "Epoch 39/100\n",
            "100/100 [==============================] - 0s 864us/step - loss: 4.1123 - mae: 1.5992\n",
            "Epoch 40/100\n",
            "100/100 [==============================] - 0s 874us/step - loss: 4.0573 - mae: 1.5848\n",
            "Epoch 41/100\n",
            "100/100 [==============================] - 0s 855us/step - loss: 4.0121 - mae: 1.5760\n",
            "Epoch 42/100\n",
            "100/100 [==============================] - 0s 830us/step - loss: 3.9760 - mae: 1.5633\n",
            "Epoch 43/100\n",
            "100/100 [==============================] - 0s 824us/step - loss: 3.9447 - mae: 1.5563\n",
            "Epoch 44/100\n",
            "100/100 [==============================] - 0s 833us/step - loss: 3.9176 - mae: 1.5482\n",
            "Epoch 45/100\n",
            "100/100 [==============================] - 0s 919us/step - loss: 3.9001 - mae: 1.5436\n",
            "Epoch 46/100\n",
            "100/100 [==============================] - 0s 875us/step - loss: 3.8820 - mae: 1.5399\n",
            "Epoch 47/100\n",
            "100/100 [==============================] - 0s 885us/step - loss: 3.8711 - mae: 1.5359\n",
            "Epoch 48/100\n",
            "100/100 [==============================] - 0s 930us/step - loss: 3.8636 - mae: 1.5358\n",
            "Epoch 49/100\n",
            "100/100 [==============================] - 0s 861us/step - loss: 3.8559 - mae: 1.5336\n",
            "Epoch 50/100\n",
            "100/100 [==============================] - 0s 857us/step - loss: 3.8508 - mae: 1.5325\n",
            "Epoch 51/100\n",
            "100/100 [==============================] - 0s 841us/step - loss: 3.8457 - mae: 1.5309\n",
            "Epoch 52/100\n",
            "100/100 [==============================] - 0s 919us/step - loss: 3.8430 - mae: 1.5318\n",
            "Epoch 53/100\n",
            "100/100 [==============================] - 0s 854us/step - loss: 3.8412 - mae: 1.5308\n",
            "Epoch 54/100\n",
            "100/100 [==============================] - 0s 848us/step - loss: 3.8387 - mae: 1.5306\n",
            "Epoch 55/100\n",
            "100/100 [==============================] - 0s 840us/step - loss: 3.8369 - mae: 1.5300\n",
            "Epoch 56/100\n",
            "100/100 [==============================] - 0s 866us/step - loss: 3.8344 - mae: 1.5305\n",
            "Epoch 57/100\n",
            "100/100 [==============================] - 0s 854us/step - loss: 3.8346 - mae: 1.5294\n",
            "Epoch 58/100\n",
            "100/100 [==============================] - 0s 877us/step - loss: 3.8333 - mae: 1.5298\n",
            "Epoch 59/100\n",
            "100/100 [==============================] - 0s 913us/step - loss: 3.8340 - mae: 1.5300\n",
            "Epoch 60/100\n",
            "100/100 [==============================] - 0s 845us/step - loss: 3.8332 - mae: 1.5299\n",
            "Epoch 61/100\n",
            "100/100 [==============================] - 0s 854us/step - loss: 3.8326 - mae: 1.5306\n",
            "Epoch 62/100\n",
            "100/100 [==============================] - 0s 852us/step - loss: 3.8315 - mae: 1.5304\n",
            "Epoch 63/100\n",
            "100/100 [==============================] - 0s 915us/step - loss: 3.8321 - mae: 1.5301\n",
            "Epoch 64/100\n",
            "100/100 [==============================] - 0s 886us/step - loss: 3.8322 - mae: 1.5304\n",
            "Epoch 65/100\n",
            "100/100 [==============================] - 0s 843us/step - loss: 3.8307 - mae: 1.5291\n",
            "Epoch 66/100\n",
            "100/100 [==============================] - 0s 843us/step - loss: 3.8308 - mae: 1.5296\n",
            "Epoch 67/100\n",
            "100/100 [==============================] - 0s 840us/step - loss: 3.8324 - mae: 1.5309\n",
            "Epoch 68/100\n",
            "100/100 [==============================] - 0s 838us/step - loss: 3.8326 - mae: 1.5308\n",
            "Epoch 69/100\n",
            "100/100 [==============================] - 0s 851us/step - loss: 3.8297 - mae: 1.5290\n",
            "Epoch 70/100\n",
            "100/100 [==============================] - 0s 977us/step - loss: 3.8314 - mae: 1.5299\n",
            "Epoch 71/100\n",
            "100/100 [==============================] - 0s 881us/step - loss: 3.8308 - mae: 1.5298\n",
            "Epoch 72/100\n",
            "100/100 [==============================] - 0s 864us/step - loss: 3.8324 - mae: 1.5301\n",
            "Epoch 73/100\n",
            "100/100 [==============================] - 0s 853us/step - loss: 3.8309 - mae: 1.5303\n",
            "Epoch 74/100\n",
            "100/100 [==============================] - 0s 858us/step - loss: 3.8334 - mae: 1.5306\n",
            "Epoch 75/100\n",
            "100/100 [==============================] - 0s 855us/step - loss: 3.8311 - mae: 1.5308\n",
            "Epoch 76/100\n",
            "100/100 [==============================] - 0s 868us/step - loss: 3.8319 - mae: 1.5307\n",
            "Epoch 77/100\n",
            "100/100 [==============================] - 0s 891us/step - loss: 3.8322 - mae: 1.5305\n",
            "Epoch 78/100\n",
            "100/100 [==============================] - 0s 847us/step - loss: 3.8307 - mae: 1.5299\n",
            "Epoch 79/100\n",
            "100/100 [==============================] - 0s 884us/step - loss: 3.8314 - mae: 1.5296\n",
            "Epoch 80/100\n",
            "100/100 [==============================] - 0s 832us/step - loss: 3.8322 - mae: 1.5299\n",
            "Epoch 81/100\n",
            "100/100 [==============================] - 0s 879us/step - loss: 3.8310 - mae: 1.5294\n",
            "Epoch 82/100\n",
            "100/100 [==============================] - 0s 909us/step - loss: 3.8314 - mae: 1.5295\n",
            "Epoch 83/100\n",
            "100/100 [==============================] - 0s 849us/step - loss: 3.8311 - mae: 1.5310\n",
            "Epoch 84/100\n",
            "100/100 [==============================] - 0s 819us/step - loss: 3.8313 - mae: 1.5291\n",
            "Epoch 85/100\n",
            "100/100 [==============================] - 0s 870us/step - loss: 3.8312 - mae: 1.5296\n",
            "Epoch 86/100\n",
            "100/100 [==============================] - 0s 836us/step - loss: 3.8319 - mae: 1.5314\n",
            "Epoch 87/100\n",
            "100/100 [==============================] - 0s 859us/step - loss: 3.8315 - mae: 1.5318\n",
            "Epoch 88/100\n",
            "100/100 [==============================] - 0s 872us/step - loss: 3.8312 - mae: 1.5305\n",
            "Epoch 89/100\n",
            "100/100 [==============================] - 0s 867us/step - loss: 3.8320 - mae: 1.5309\n",
            "Epoch 90/100\n",
            "100/100 [==============================] - 0s 860us/step - loss: 3.8315 - mae: 1.5304\n",
            "Epoch 91/100\n",
            "100/100 [==============================] - 0s 868us/step - loss: 3.8322 - mae: 1.5298\n",
            "Epoch 92/100\n",
            "100/100 [==============================] - 0s 895us/step - loss: 3.8313 - mae: 1.5301\n",
            "Epoch 93/100\n",
            "100/100 [==============================] - 0s 845us/step - loss: 3.8314 - mae: 1.5305\n",
            "Epoch 94/100\n",
            "100/100 [==============================] - 0s 881us/step - loss: 3.8313 - mae: 1.5307\n",
            "Epoch 95/100\n",
            "100/100 [==============================] - 0s 878us/step - loss: 3.8316 - mae: 1.5303\n",
            "Epoch 96/100\n",
            "100/100 [==============================] - 0s 909us/step - loss: 3.8332 - mae: 1.5302\n",
            "Epoch 97/100\n",
            "100/100 [==============================] - 0s 856us/step - loss: 3.8320 - mae: 1.5301\n",
            "Epoch 98/100\n",
            "100/100 [==============================] - 0s 858us/step - loss: 3.8326 - mae: 1.5309\n",
            "Epoch 99/100\n",
            "100/100 [==============================] - 0s 848us/step - loss: 3.8319 - mae: 1.5317\n",
            "Epoch 100/100\n",
            "100/100 [==============================] - 0s 840us/step - loss: 3.8326 - mae: 1.5298\n",
            "w =  [[1.99320257]\n",
            " [-0.999715507]]\n",
            "b =  [3.08692026]\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "If using Keras pass *_constraint arguments to layers.\n",
            "INFO:tensorflow:Assets written to: ./data/linear_model/1/assets\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NvTrC280nTui",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5578afc3-bbcb-44ea-8bfb-d238a3801aaa"
      },
      "source": [
        "#查看保存的模型文件\n",
        "!ls {export_path+version}"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "assets\tsaved_model.pb\tvariables\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FDyaZEudnaDn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a4671ead-7be0-4232-ef31-8cbfdbc51372"
      },
      "source": [
        "# 查看模型文件相关信息\n",
        "!saved_model_cli show --dir {export_path+str(version)} --all"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:\n",
            "\n",
            "signature_def['__saved_model_init_op']:\n",
            "  The given SavedModel SignatureDef contains the following input(s):\n",
            "  The given SavedModel SignatureDef contains the following output(s):\n",
            "    outputs['__saved_model_init_op'] tensor_info:\n",
            "        dtype: DT_INVALID\n",
            "        shape: unknown_rank\n",
            "        name: NoOp\n",
            "  Method name is: \n",
            "\n",
            "signature_def['serving_default']:\n",
            "  The given SavedModel SignatureDef contains the following input(s):\n",
            "    inputs['inputs'] tensor_info:\n",
            "        dtype: DT_FLOAT\n",
            "        shape: (-1, 2)\n",
            "        name: serving_default_inputs:0\n",
            "  The given SavedModel SignatureDef contains the following output(s):\n",
            "    outputs['outputs'] tensor_info:\n",
            "        dtype: DT_FLOAT\n",
            "        shape: (-1, 1)\n",
            "        name: StatefulPartitionedCall:0\n",
            "  Method name is: tensorflow/serving/predict\n",
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0406 13:11:10.866565 139649829848960 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling __init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "If using Keras pass *_constraint arguments to layers.\n",
            "\n",
            "Defined Functions:\n",
            "  Function Name: '__call__'\n",
            "    Option #1\n",
            "      Callable with:\n",
            "        Argument #1\n",
            "          inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name=u'inputs')\n",
            "        Argument #2\n",
            "          DType: bool\n",
            "          Value: False\n",
            "        Argument #3\n",
            "          DType: NoneType\n",
            "          Value: None\n",
            "    Option #2\n",
            "      Callable with:\n",
            "        Argument #1\n",
            "          inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name=u'inputs')\n",
            "        Argument #2\n",
            "          DType: bool\n",
            "          Value: True\n",
            "        Argument #3\n",
            "          DType: NoneType\n",
            "          Value: None\n",
            "\n",
            "  Function Name: '_default_save_signature'\n",
            "    Option #1\n",
            "      Callable with:\n",
            "        Argument #1\n",
            "          inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name=u'inputs')\n",
            "\n",
            "  Function Name: 'call_and_return_all_conditional_losses'\n",
            "    Option #1\n",
            "      Callable with:\n",
            "        Argument #1\n",
            "          inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name=u'inputs')\n",
            "        Argument #2\n",
            "          DType: bool\n",
            "          Value: False\n",
            "        Argument #3\n",
            "          DType: NoneType\n",
            "          Value: None\n",
            "    Option #2\n",
            "      Callable with:\n",
            "        Argument #1\n",
            "          inputs: TensorSpec(shape=(None, 2), dtype=tf.float32, name=u'inputs')\n",
            "        Argument #2\n",
            "          DType: bool\n",
            "          Value: True\n",
            "        Argument #3\n",
            "          DType: NoneType\n",
            "          Value: None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GffjP0o_nbRy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vsi0qz-zndv4",
        "colab_type": "text"
      },
      "source": [
        "### 二，安装 tensorflow serving\n",
        "\n",
        "安装 tensorflow serving 有2种主要方法：通过Docker镜像安装，通过apt安装。\n",
        "\n",
        "通过Docker镜像安装是最简单，最直接的方法，推荐采用。\n",
        "\n",
        "Docker可以理解成一种容器，其上面可以给各种不同的程序提供独立的运行环境。\n",
        "\n",
        "一般业务中用到tensorflow的企业都会有运维同学通过Docker 搭建 tensorflow serving.\n",
        "\n",
        "无需算法工程师同学动手安装，以下安装过程仅供参考。\n",
        "\n",
        "不同操作系统机器上安装Docker的方法可以参照以下链接。\n",
        "\n",
        "Windows: https://www.runoob.com/docker/windows-docker-install.html\n",
        "\n",
        "MacOs: https://www.runoob.com/docker/macos-docker-install.html\n",
        "\n",
        "CentOS: https://www.runoob.com/docker/centos-docker-install.html\n",
        "\n",
        "安装Docker成功后，使用如下命令加载 tensorflow/serving 镜像到Docker中\n",
        "\n",
        "docker pull tensorflow/serving"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "38V2IWzAnfsy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_5Gd3NayoQ7P",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "4201d34c-01bb-4cba-df94-124f194a2292"
      },
      "source": [
        "!curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100  2943  100  2943    0     0  15248      0 --:--:-- --:--:-- --:--:-- 15248\n",
            "OK\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fnX4sHQhoch4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 530
        },
        "outputId": "03000e78-848d-4653-8c37-8d8f5fdad84b"
      },
      "source": [
        "!apt update"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[33m\r0% [Working]\u001b[0m\r            \rGet:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n",
            "\u001b[33m\r0% [Connecting to archive.ubuntu.com (91.189.88.142)] [1 InRelease 2,586 B/88.7\u001b[0m\u001b[33m\r0% [Waiting for headers] [Waiting for headers] [Waiting for headers] [Waiting f\u001b[0m\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Waiting for headers] [Wait\u001b[0m\r                                                                               \rGet:2 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ InRelease [3,626 B]\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [2 InRelease 3,626 B/3,626 \u001b[0m\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Waiting for headers] [Wait\u001b[0m\r                                                                               \rIgn:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Waiting for headers] [Wait\u001b[0m\r                                                                               \rHit:4 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Waiting for headers] [Conn\u001b[0m\r                                                                               \rHit:5 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Waiting for headers] [Conn\u001b[0m\r                                                                               \rIgn:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Connecting to ppa.launchpa\u001b[0m\r                                                                               \rHit:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Connecting to ppa.launchpa\u001b[0m\r                                                                               \rGet:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [8 Release 564 B/564 B 100%\u001b[0m\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [Waiting for headers] [Connecting to ppa.launchpa\u001b[0m\r                                                                               \rGet:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [9 InRelease 2,601 B/88.7 kB 3%] [Waiting for hea\u001b[0m\r                                                                               \rGet:10 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]\n",
            "\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [9 InRelease 2,601 B/88.7 kB 3%] [10 Release.gpg \u001b[0m\u001b[33m\r0% [1 InRelease gpgv 88.7 kB] [9 InRelease 2,601 B/88.7 kB 3%] [Waiting for hea\u001b[0m\u001b[33m\r                                                                               \r0% [9 InRelease 15.6 kB/88.7 kB 18%] [Waiting for headers]\u001b[0m\u001b[33m\r0% [2 InRelease gpgv 3,626 B] [9 InRelease 15.6 kB/88.7 kB 18%] [Waiting for he\u001b[0m\r                                                                               \rGet:11 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [37.0 kB]\n",
            "\u001b[33m\r0% [2 InRelease gpgv 3,626 B] [9 InRelease 15.6 kB/88.7 kB 18%] [11 Packages 4,\u001b[0m\u001b[33m\r0% [2 InRelease gpgv 3,626 B] [9 InRelease 21.4 kB/88.7 kB 24%] [Waiting for he\u001b[0m\r                                                                               \rGet:12 http://ppa.launchpad.net/marutter/c2d4u3.5/ubuntu bionic InRelease [15.4 kB]\n",
            "\u001b[33m\r                                                                               \r0% [2 InRelease gpgv 3,626 B] [9 InRelease 21.4 kB/88.7 kB 24%]\u001b[0m\u001b[33m\r0% [11 Packages store 0 B] [2 InRelease gpgv 3,626 B] [9 InRelease 21.4 kB/88.7\u001b[0m\u001b[33m\r0% [2 InRelease gpgv 3,626 B] [9 InRelease 24.3 kB/88.7 kB 27%] [Waiting for he\u001b[0m\r                                                                               \rGet:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [870 kB]\n",
            "Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [835 kB]\n",
            "Get:15 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ Packages [87.7 kB]\n",
            "Get:16 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n",
            "Get:18 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages [31.7 kB]\n",
            "Get:19 http://ppa.launchpad.net/marutter/c2d4u3.5/ubuntu bionic/main Sources [1,809 kB]\n",
            "Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [12.2 kB]\n",
            "Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,367 kB]\n",
            "Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,168 kB]\n",
            "Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [53.3 kB]\n",
            "Get:24 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,247 B]\n",
            "Get:25 http://ppa.launchpad.net/marutter/c2d4u3.5/ubuntu bionic/main amd64 Packages [873 kB]\n",
            "Fetched 7,420 kB in 3s (2,288 kB/s)\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "61 packages can be upgraded. Run 'apt list --upgradable' to see them.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0CclAJ3Noc2F",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 445
        },
        "outputId": "05e4ada5-7b99-44d4-d3df-02e1b6aa4b6f"
      },
      "source": [
        "!echo \"deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\" | tee /etc/apt/sources.list.d/tensorflow-serving.list && \\\n",
        "curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -\n",
        "!apt update"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100  2943  100  2943    0     0   106k      0 --:--:-- --:--:-- --:--:--  106k\n",
            "OK\n",
            "Get:1 http://storage.googleapis.com/tensorflow-serving-apt stable InRelease [3,012 B]\n",
            "Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease\n",
            "Hit:3 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/ InRelease\n",
            "Hit:4 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease\n",
            "Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
            "Get:6 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server-universal amd64 Packages [361 B]\n",
            "Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease\n",
            "Hit:8 http://ppa.launchpad.net/marutter/c2d4u3.5/ubuntu bionic InRelease\n",
            "Get:9 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages [354 B]\n",
            "Hit:10 http://archive.ubuntu.com/ubuntu bionic-backports InRelease\n",
            "Ign:11 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
            "Ign:12 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
            "Hit:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release\n",
            "Hit:14 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n",
            "Fetched 3,727 B in 2s (2,158 B/s)\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "61 packages can be upgraded. Run 'apt list --upgradable' to see them.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LYDySaUjopsT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 292
        },
        "outputId": "061ae8fe-5310-45e2-ab8b-ec6977674fd1"
      },
      "source": [
        "!apt-get install tensorflow-model-server"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following NEW packages will be installed:\n",
            "  tensorflow-model-server\n",
            "0 upgraded, 1 newly installed, 0 to remove and 61 not upgraded.\n",
            "Need to get 175 MB of archives.\n",
            "After this operation, 0 B of additional disk space will be used.\n",
            "Get:1 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 tensorflow-model-server all 2.1.0 [175 MB]\n",
            "Fetched 175 MB in 3s (68.2 MB/s)\n",
            "Selecting previously unselected package tensorflow-model-server.\n",
            "(Reading database ... 133872 files and directories currently installed.)\n",
            "Preparing to unpack .../tensorflow-model-server_2.1.0_all.deb ...\n",
            "Unpacking tensorflow-model-server (2.1.0) ...\n",
            "Setting up tensorflow-model-server (2.1.0) ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VNx_ehhvop70",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BZGSd5jDnhB5",
        "colab_type": "text"
      },
      "source": [
        "### 三，启动 tensorflow serving 服务"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8unJMNIkoiB9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a849fe75-2478-4a35-edfc-e946303d4dda"
      },
      "source": [
        "import os\n",
        "os.environ[\"MODEL_DIR\"] = \"/content/data/linear_model/\"\n",
        "os.environ[\"MODEL_DIR\"]"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'/content/data/linear_model/'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3gGy_MtHoiZX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8f8f57eb-07a2-4e20-fc0f-35ea0277afd5"
      },
      "source": [
        "!ls /content/data/linear_model/"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hZGwhUMeo1wC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "bfa05e91-02d9-430d-feb5-b82ab9d302c2"
      },
      "source": [
        "%%bash --bg \n",
        "nohup tensorflow_model_server \\\n",
        "  --rest_api_port=8501 \\\n",
        "  --model_name=linear_model \\\n",
        "  --model_base_path=\"${MODEL_DIR}\" >server.log 2>&1"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Starting job # 0 in a separate thread.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wm6gHMIuo2OU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "f6bbebea-28c9-4945-eb9a-b5690c82a562"
      },
      "source": [
        "# 查看日志，有可能会有报错信息\n",
        "!tail server.log"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[warn] getaddrinfo: address family for nodename not supported\n",
            "[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cVFSDfUFpEho",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0OfWKzFpMvg",
        "colab_type": "text"
      },
      "source": [
        "### 四，向服务API发送请求\n",
        "可以使用任何编程语言的http功能发送请求，下面示范linux的 curl 命令发送请求，以及Python的requests库发送请求。\n",
        "\n",
        "使用 Linux 的curl命令发送请求"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O1J4PRRsniau",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "afe51dac-1bd8-47bf-e699-1aa0215df665"
      },
      "source": [
        "!curl -d '{\"instances\": [[1.0, 2.0], [5.0,8.0]]}' \\\n",
        "    -X POST http://localhost:8501/v1/models/linear_model:predict"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"predictions\": [[3.08069181], [5.05520916]\n",
            "    ]\n",
            "}"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qdpY1wBDpgTH",
        "colab_type": "text"
      },
      "source": [
        "利用 Python request库发送请求"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J9x-p4mUnjyA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e5240ef2-a487-4ea7-c2e5-82f9f0567f8a"
      },
      "source": [
        "import json\n",
        "data = json.dumps({\"signature_name\": \"serving_default\", \"instances\": [[1.0, 2.0], [5.0,7.0]]})\n",
        "\n",
        "import requests\n",
        "headers = {\"content-type\": \"application/json\"}\n",
        "json_response = requests.post('http://localhost:8501/v1/models/linear_model:predict', data=data, headers=headers)\n",
        "predictions = json.loads(json_response.text)[\"predictions\"]\n",
        "print(predictions)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[3.08069181], [6.05492449]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KOe-AOrNnqGI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}